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Prioritising Data Elements of a Data Stream 
Field of the invention 

5 

The present invention relates to prioritising data elements 
of a data stream for transmission to a receiving device. 

Background of the invention 

10 

With the increased proliferation of data processing devices 
and with increased availability of computer networks for data 
exchange an increasing number of services can now be obtained 
using home computer equipment or similar. 

15 

Generally, the provision of a service via a computer network 
requires a transmission device or server for transmitting 
data pertaining to the service and requires a receiving 
device or client for receiving the data from the server. A 
2 0 user operating the receiving device may then appropriately 

take notice of the received data, e.g. view the received data 
on a display, or listen to audio data. Interactive services 
enable a user to control the service provision via entering 
information at the client for transmission to the server, 

2 5 enabling the server to appropriately configure the service 

provision . 

A service provided over a computer network may require the 
exchange of substantial amounts of data, for example image or 

3 0 audio data, and, as the network bandwidth is generally 

limited, data compression or coding techniques have been 
developed. These techniques reduce the amount of data to be 
transmitted via the network, preferably without substantially 
degrading the quality of the data obtained at the receiver. 
3 5 Many data compression techniques for image data, including 
video, and audio data are available today. 
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One group of services available over computer networks 
includes the transmission of data streams from a transmitting 
device to a receiving device. The data stream may incorporate 
a sequence of images, such as a video sequence, or may 
5 incorporate audio data, or both. For example, a user at a 
client computer device may obtain a steaming service by 
appropriately selecting an offered data stream, e.g. by 
clicking on a corresponding icon on a browser screen. The 
user could view a browser page on the client display 

10 including for example a selection of text elements 

accompanied by a tag offering a corresponding video. The user 
can then select a desired tag and a selection instruction 
will be transmitted to a server, requesting the server to 
initiate transmission of a corresponding video data stream to 

15 the client device. The client device in turn will display the 
video sequence. 

As a data stream usually contains a large amount of data to 
be transmitted under relatively strict time constrains, e.g. 

2 0 for maintaining real time requirements, efficient data 

compression techniques or codecs are required. 

Streaming applications generally involve the use of 
predictive coding. A data stream is usually formed by the 
25 sequential transmission of a number of data elements, each of 
the data elements representing a portion of the information 
to be steamed to the receiving device. Predictive coding 
includes any data compression or coding technique, where at 
least some of the data elements refer to or require 

3 0 information transmitted in another data element. A data 

element requiring information from another data element can 
therefore be decompressed at the receiving device only to 
full extent, if the required further data element or data 
elements are available at the receiving device. For example, 
35 predictive coding may include a statistical estimation 

procedure where future random variables of the data stream 
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are estimated or predicted from past and present observable 
random variables. 

Streaming applications between computers may be hampered by a 
5 lack of service support. Generally it is difficult to reserve 
a bandwidth, e.g. on a computer network, for the streaming 
application and for this reason a transmitting device may be 
unable to transmit the complete streaming information to the 
receiving device in a timely manner. If not all data elements 
10 of the streaming application can be transmitted in a timely 
manner to the receiving device, a video, audio or other data 
stream may be interrupted or fully terminated. 

It is therefore required to appropriately scale down a 
3_5 required bandwidth for the data steam transmission while 

accepting a more or less severe reduction of the quality of 
the stream but avoiding dropouts or termination of services. 

One approach to scale the required bandwidth of a data stream 

2 0 is stream switching was proposed in "Video coding for 

multiple target audiences" by A. Lipman in Proc . Visual 
Communications and Image Processing, San Jose, CA, USA, 
January 1999, pages 780-782. Here a server dynamically 
switches between different streams representing the same 
2.5 video stream in different qualities and thus with different 
bandwidth requirements. However, the granularity of the 
adjustment is coarse and a large number of streams must be 
stored . 

3 0 A block-based hybrid video coder capable of providing a more 

granular scalability by supporting spatial, signal to noise 
and temporal scalability and combined methods are described 
in reference to M. Domanski , S. Mackowia, "Modified mpeg-2 
video coders with efficient multi-layer scalability", in 
3 5 Proc. IEEE International Conference on Image Processing, 

(ICIP 2001), Thessaloniki, October 2001, pp. 1003-36. Here, a 
video stream is subdivided into several layers of increasing 
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detail and the number of transmitted layers can be varied. 
However, the overhead may not be acceptable. 

A further concept to adaptively reduce a bandwidth 
5 requirement of a data stream is proposed in reference S . Kang 
and A. Zakhor, "Packet scheduling algorithm for wireless 
video streaming" in Proc. of the 12th Packet Video Workshop 
2002, Pittsburg, PA, April 2002. In this concept the 
dependencies between individual frames constituting the data 
10 stream are taken into account, and, frames depending on a 
reference frame, which was not transmitted, are dropped. 

Another concept for adapted bandwidth reduction was proposed 
in P. Chou and Z. Miao, "Rate-distortion optimised streaming 
15 of packetized media," Microsoft Research, Techn. Rep. MSR-TR- 
2001-35, February 2001, optimising a peak signal to noise 
ratio of a video stream by dynamically assigning priority 
values to frames constituting the data stream and therefore 
quantifying the importance. 

20 

However, none of the above yields satisfactory results under 
all operating conditions. 

Summary of the invention 

25 

It is therefore an object of the invention to provide for 
improved priority assignment to data elements of a data 
stream for transmission from a transmitting device to a 
receiving device. 

30 

A transmission device for prioritising data elements of a 
data stream for transmission to a receiving device comprises 
decodability determining means for determining a decodability 
of a current data element, the decodability indicating the 
3 5 extent to which the current data element is decodable at the 
receiving device; prioritising means for assigning a priority 
to the current data element based on the determined 
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decidability; and a transmitter controller for scheduling a 
transmission of the current data element to the receiving 
device based on the priority. Thus, the invention enables 
employing the decodability of a data element at the receiving 
5 device for determining a priority of the data element to be 
used when scheduling the transmission. 

According to advantageous embodiment the decodability 
determining means is adapted to determine the decodability of 
10 the current data element using information on which of a 

plurality of data elements were transmitted to the receiving 
device. Thus, the transmission device may employ information 
on the availability of data elements and the receiving device 
for determining a decodability of a data element . 

15 

According to another embodiment the decodability determining 
means receives a feedback from the receiving device, the 
feedback indicating which of the data elements were received 
error- free. Accordingly, the transmission device may employ 
2 0 example acknowledgements in a packet or block transmission 
system for obtaining information on which of the data 
elements were successfully transmitted to the receiving 
device . 

2 5 According to another embodiment, if the current data element 

requires a reference data element for being fully decodable 
at the receiving device, the decodability of the current data 
element is set equal to the decodability of the reference 
data element, when the second data element has been 

3 0 transmitted. Similarly, if the current data element requires 

multiple reference data elements for being fully decodable at 
the receiving device, the decodability of the current data 
element is determined based on the decodabilities of the 
reference data elements. Thus, it is possible to determine a 
35 decodability in cases of multiple dependencies between 
different data elements. 
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According to another embodiment the decodability determining 
means is adapted to recalculate the decodability of at least 
a portion of the data elements upon transmission of the 
current data element. Preferably, the decodability of such 
5 data elements is recalculated, which are indicated m a 

decoding dependency record of the current data element, the 
decoding dependency record indicating all data elements 
requiring the current data element for decoding. Thus, the 
transmission device may dynamically update the decodabilities 
10 of the data elements, and may reduce complexity of the 
updating procedure, by referring to decoding dependency 
records . 

According to another embodiment the data stream may comprise 
15 independent data elements being independent from other data 
elements and dependent data elements being dependent on at 
least one reference data element, wherein the decodability of 
an independent data element is set to a maximum decodability 
and the decodability of a dependent data element is set equal 
2 0 to the decodability of a reference data element, when the 

reference data element has been transmitted. Accordingly, the 
transmission device may efficiently handle data streams 
comprising independent and dependent data elements. 

2 5 According to another embodiment, the data stream comprises 

intra -blocks being independent from other data elements and 
the decodability of an intra-block being set to a maximum 
decodability, indicating that the intra-block is fully 
decodable at the receiving device; inter-blocks encoding 

3 0 differences between content of data element and content of a 

reference data element, the decodability of an inter-block 
being set equal to the decodability of the reference data 
block; and skip-blocks indicating content requiring content 
of a reference data block, the decodability of an inter-block 
3 5 being set equal to the decodability of the reference data 
block. Thus, for example in video streaming applications, 
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decodability based prioritising of transmission blocks may 
advantageously be employed. 

According to another embodiment, the decodability determining 
5 means is adapted determine an average decodability of a 

number of data elements; to determine a decodability increase 
of the average decodability of the number of data elements 
obtainable by transmitting the current data elements; and to 
determine the priority of the current data element based on 
10 the decodability increase. Accordingly, the overall benefit 
of the transmission of a data element may be employed for 
assigning a priority to a data element. 

According to another embodiment the prioritising means may be 
15 adapted to determine the priority of the current data element 
based on the decodability increase divided by the size of the 
current data element. Accordingly, a packet size may be used 
to further improve the priority assigning process. 

2 0 According to another embodiment the number of data elements 

represents data elements of a predetermined time window of 
the data stream or of the entire data stream. 

According to another embodiment the transmitter controller is 
25 adapted to estimate the probability of an error- free 

transmission of a transmission channel. Thus, in the absence 
for example of an acknowledgement of received error- free 
packet at the receiving device, the transmitting device may 
estimate the error probability of the transmission channel, 

3 0 e.g. for use in determining a decodability. 

According to another embodiment, the decodability determining 
means determine the decodability as a product of the extent 
to which the current data element is decodable at the 
35 receiving device and the probability of an error- free 

transmission of the current data packet over a transmission 
channel. Further, the decodability of an independent data 
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element may be set to be equal to the determined probability, 
and the decodability of a dependent data element may be set 
equal to the determined probability times the decodability 
determine based on the decodability of the at least one 
5 reference data element . 

According to another embodiment, the decodability determining 
means sets the decodability of an intra-block equal to the 
probability of an error-free transmission, sets the 
10 decodability of an inter-block equal to the decodability of 
the reference data block multiplied by the probability of an 
error- free transmission; and the decodability of a skip block 
independent of the probability of an error- free transmission. 

15 According to another embodiment, the decodability determining 
means is adapted to set the decodability of the reference 
data element equal to the decodability of a data frame 
containing the reference data element . 

2 0 Further, if the data stream is a video stream, for reducing a 

complexity, motion compensation for determining a 
decodability may be disregarded. 

According to another embodiment of the invention, a 
25 transmission method for prioritising data elements of a data 
stream for transmission to a receiving device may comprise 
determining a decodability of a current data element, the 
decodability indicating the extent to which the current data 
element is decodable at the receiving device; assigning a 

3 0 priority to the current data element based on the determined 

decodability; and scheduling a transmission of the current 
data element to the receiving device based on the priority. 



Further features of the invention are disclosed in further 
3 5 claims . 
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Brief description of the drawings 

Fig. 1 illustrates elements of a transmission system 
5 according to an embodiment of the invention; 

Fig. 2 illustrates elements of a method for prioritising 

data elements of a data stream for transmission to 
a receiving device according to an embodiment of 
10 the invention; 

Fig. 3 illustrates operations of a method for prioritising 

data elements for transmission to a receiving 
device according to another embodiment of the 
15 . invention, particularly outlining operations for 

using information on data elements which have been 
transmitted; 

Fig. 4 illustrates operations of a method for prioritising 

2 0 data elements for transmission to a receiving 

device according to another embodiment of the 
invention, particularly illustrating operations for 
recalculating a decodability of data elements; 

25 Fig. 5 illustrates operations of a method for prioritising 

data elements for transmission to a receiving 
device according to another embodiment of the 
invention, particularly illustrating operations for 
employing a decodability increase for assigning a 

3 0 priority ; 

Fig. 6 illustrates operations of a method for prioritising 

data elements of a data stream for transmission to 
a receiving device according to another embodiment 
3 5 of the invention, particularly illustrating 

operations for handling independent and dependent 
data elements; 
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Fig. 7 illustrates operations for prioritising data 

elements for transmission to a receiving device 
according to another embodiment of the invention, 
5 particularly illustrating determining 

decodabilities of independent and dependent data 
elements ; 

Fig. 8 illustrates operations of a method for prioritising 
10 data elements for transmission to a receiving 

device, particularly illustrating operations for 
handling intra-blocks , inter-blocks and skip- 
blocks, e.g. of a video stream; 

15 Fig. 9 illustrates a state diagram of the expected error 

without forward error control; 

Fig. 10 illustrates a state diagram of the change of the 

expected error ; 

20 

Fig. 11 illustrates a decodability of frames using constant 

parameters with a group of frame length 15; and 

Fig. 12 illustrates a priority of frames using constant 
25 parameters. 

Detailed description of the preferred embodiments 



In the following an embodiment of the invention will be 
30 described with regard to Fig. 1. 

Fig. 1 schematically illustrates elements of a system for 
prioritising data elements for transmission from a 
transmitting device to a receiving device including 
35 determining decodabilities of data elements, assigning 
priorities to the data elements based on the determined 
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decodabilities and scheduling a transmission of the data 
elements to the receiving device based on the priority. 

Fig. 1 depicts a transmission device denoted 100 for 
5 prioritising data elements of a data stream for transmission 
to a receiving device 110. A data element transmission 
performed in accordance with the assigned priorities from the 
transmitting device to the receiving device is illustrated at 
150 . 

* 

10 

The transmitting device 100 comprises decodability 
determining means 101 for determining a decodability of a 
current data element, the decodability indicating the extent 
to which the current data element is decodable at the 
15 receiving device. Further, the transmission device comprises 
prioritising means 102 for assigning a priority to the 
current data element based on the determined decodability, 
and, the transmission device comprises a transmitter 
controller 103 for scheduling a transmission of the current 

2 0 data element to the receiving device based on the priority. 

The transmitting device thus allows considering information 
on the decodability of a data element at the receiving 
device, when assigning priorities to the data elements for 
planning a transmission. 

25 

If a predictive coding scheme is used for transmitting the 
data stream from the transmitting device to the receiving 
device, the extent to which a data element is decodable at 
the receiving device may depend on the availability of 

3 0 another data element at the receiving device. If this other 

data element is available at the receiving device, the 
current data element may be fully decodable, whereas, if the 
other data element is unavailable at the receiving device, 
the current data element may be not or not fully decodable at 
35 the receiving device. In an example the decodability 

determining means thus determines the decodability of the 
current data element using information on which of the data 
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elements of the stream where already transmitted to the 
receiving device, allowing a prediction, whether the a 
currently transmitted data element is decodable. 

5 The information on the transmission of the data elements to 
the receiving device may be obtained from the transmitter 
controller 103 , i.e., the decodability determining means 101 
may be informed by the transmitter controller 103 regarding 
the identity of the data elements which were sent via a 

10 communication channel towards the receiving device. Based on 
the information which data elements have been scheduled for 
transmission at the transmitter controller an placed onto the 
communication channel, the decodability determining means may 
deduce availability of the data element at the receiving 

15 device after a certain time required for transmission, and 
assuming an error free transmission channel. 

Alternatively, the transmitter controller or directly the 
decodability determining means may receive a feedback from 

2 0 the receiving device indicating which of the data elements 

were received error free. This information may be returned to 
the transmitting device via an acknowledgement message, as 
known in the art. Having confirmed information on which data 
elements have been received error free at the receiving 
25 device, the determination of the decodability of a current 
data element can be improved. 

The prioritising means receives the determined decodability 
of the data elements and assigns a priority to the data 

3 0 elements and assigns a priority to the data elements based 

thereon. In the simplest case the priority may directly 
correspond to the determined decodability of data element 
under consideration. Further, the prioritising means may use 
the nature of the data element under consideration. For 
3 5 example, a data element may be of high importance for the 

overall data stream or may include important information for 
a user obtaining the data stream. In this case the 
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prioritising means will assign a high priority to such a data 
element. On the other hand, a data element may not be 
essential for the transmission of the overall data stream 
and/or may not be containing highly important information for 
5 a user, and in this case the prioritising means may assign a 
lower priority to such data elements. In an example the 
prioritising means determines the priority of a data element 
by multiplying the determined decodability of the data 
element with the importance of the data element for the 
10 physical transmission of the data stream or for the user. Of 
course, any other arithmetic combination of the importance of 
the data element and the decodability of the data element may 
be used. 

15 When handling a data stream having multiple elements, the 
result of the above operations carried out by decodability 
determining means and the prioritising means will be a set of 
elements for transmission to the receiving device, the 
elements each having respective priorities assigned thereto. 

2 0 These data elements with the associated priorities are then 

handed over to the transmitter controller, for enabling the 
transmitter controller to schedule the transmission of the 
data elements under consideration of their priority. 
Preferably, the transmitter controller will transmit data 
25 elements with the highest priority at an earliest point in 
time, followed by data elements with gradually lower 
priority. 

According to the above scheme, in a streaming application 

3 0 highest emphasis will be put on the transmission of data 

elements having the highest priority, as they will at least 
to a high degree by decodable and as they are of high 
importance to the overall data stream, er having 

transmitted all data packets currently present with the 
35 highest priority, the transmission scheduler will turn to 
data packets having gradually lower priorities. Thus, the 
transmission scheduler will then transmit data elements with 
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the second highest priority, then with the third highest 
priority and so on, until all data packets are transmitted, 
or, until the available bandwidth is exhausted and no further 
packets can be transmitted. In this case the transmission 
5 scheduler drops all packets with this priority class and even 
lower priority, as they cannot be accommodated in the 
available bandwidth. 

As the decodabilities and the priorities preferably are 
10 subject to updating within certain time periods, during 

ongoing transmission of data elements of the data stream, in 
an example the transmitter controller handles data elements 
with constantly updated priorities, e.g. with data elements 
being upgraded to highest priority or with new data elements 
15 of highest priority or any other priority arriving from the 
prioritising means. As the transmitter controller, upon 
encountering data elements with highest priority will always 
transmit such an element first and then will transmit data 
elements with gradually lower priority, it will be assured 
2 0 that the data stream is always transmitted under adaptation 
to a varying available bandwidth. Upon reaching a bandwidth 
limit, data elements of lower priorities are dropped, but as 
soon as larger bandwidth becomes available, the transmitter 
controller will be able to also transmit data elements of 

2 5 lower priorities, thus adapting the data stream transmission 

to increase available bandwidth. 

In an example the transmitting device handles a continuous 
stream of data elements of the data stream, arriving or being 

3 0 retrieved from a buffer (not shown) at a for example constant 

rate for real time streaming, and accordingly, the 
decodability determining means, the prioritising means and 
the controller continuously operate to determine the 
decodability of data elements, to prioritise data elements 
3 5 and to transmit data elements according to priority. 
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Alternatively, however, it is also possible that the 
transmitting device treats groups of data elements as 
separate entities, for example if there are no 

interdependency between data elements of certain sets of data 
5 elements. The rate by which the transmitting device is 

handling data elements may be adjusted to match the playback 
speed of data elements of the receiving device, so that an 
accumulation of excessive amounts of data at the transmitting 
device or receiving device can be avoided, as known in the 
10 art. 

In the following- examples of the elements illustrated in Fig. 
1 will be outlined in further detail . It is noted that the 
following description is by way of example only and should 
15 not be construed, as limiting the application. 

The transmitting" device 10 0 may generally be any kind of data 
processing device with the capability of providing a data 
stream to a receiving device or a number of receiving 

2 0 devices. For example, the transmitting device may be a server 
connected to a computer network, and be adapted to provide 
one or a plurality of different data streams to one or a 
plurality of different receiving devices or clients. In an 
example the transmitting device is a web server, capable of 

2 5 providing content to clients upon request, for example by 
using HTML or XML data formats or any other kind of 
communication protocols. Preferably, the transmitting device 
100 comprises a large data store, or is connected to such a 
large data store or database for having access to information 

30 to be streamed to users. Further, the transmitting device 

preferably comprises a central processing unit or a plurality 
of central processing units adapted to react to client 
requests and to provide data streams via the network to the 
clients . 

35 

A data stream may be any kind of data collection transmitted 
to a receiving device. The data stream may comprise video 
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information, audio information, combinations of both or any 
other kind of data, for example complex 3 -dimensional 
rendered objects or similar. 

5 To reduce the required bandwidth for transmitting the data 
stream any kind of coding or compression technique may be 
applied, such as compression techniques of the H.263 or MPEG 
standards, or any other kind of streaming scheme providing a 
sequence of data elements with interdependencies . 

10 

The decodability determining means 101, the prioritising 
means 102 and the transmitter controller may be at least 
partially realized in software, with corresponding code 
instructions of the decodability determining means, the 

15 prioritising means, and the transmitter controller being 
stored in one or more memories, accessible by the central 
processing unit of the transmitting device so that the 
central processing unit is enabled to retrieve the respective 
coded instructions to execute the functionality of the 

20 decodability determining means, the prioritising means and 
the transmitter controller as detailed above. 

Alternatively, or in addition thereto the above elements may 
at least partially be realized as dedicated hardware 
25 circuits. As one example, the transmitter controller may 
comprise a codec or compression unit, constituted by a 
hardware circuit, which handles the computationally intense 
operations of the coding or compression of information of the 

» 

data stream. 

30 

The data elements may be any collection of a portion of the 
data of the data stream. For example, the data stream may be 
partitioned into data elements with the same or variable size 
during a coding or compression operation at the transmitting 
35 device. Alternatively, the data stream may be partitioned 
into data elements before or after a coding or compression 
operation. The data elements may have different types of 
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content, e.g. data describing the structure of the data 
stream or data for presentation to a user. Moreover, the data 
elements may be grouped into larger entities, such as frames, 
data packets for transmission via a packet switched network, 
5 and similar. 

The transmission of the data elements from the transmitting 
device 100 to the receiving device 110 illustrated in Fig. 1 
at 150 may be accomplished via any kind of transmission 

10 medium, including dedicated communication links, 

connectionless connections, computer networks, wireline or 
wireless links and similar. For example, the transmission may 
involve the Internet, local area networks, ad-hoc networks, 
wireless networks such as according to the GSM, UMTS , D-AMPS, 

15 3G standards and similar, alone or in combination. 

According to another example, the streaming service can be 
provided in a client server environment, the server streaming 
the data stream to the client or a plurality of clients. 
2 0 According to an alternative a proxy is provided between the 
server and the client, for delivering the data stream to the 
client (s), e.g. with a. high bandwidth transmission capability 
between the server and the proxy and a lower bandwidth 
transmission between the proxy and the client. In this case 

2 5 the proxy may at least provide some of the above and below 

described functionalities for prioritising the data elements 
of the data stream for- transmission to a receiving device, 
i.e. to at least some extent assume the functionalities of 
the transmission device. However, it is also possible that 

3 0 the proxy only forwards data elements appropriately handled 

by the sever acting at least partially as the transmission 
device . 

In the following a further embodiment of the invention will 
3 5 be described with regard to Fig. 2. 
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Fig. 2 illustrates operations of a method for prioritising 
data elements of a data stream for transmission to a 
receiving device and the operations of Fig. 2 may be carried 
using the hardware arrangement showing in Fig. 1 however, not 
5 being limited thereto. 

In a first operation 201 in the decodability of a currently 
considered data element is determined, the decodability 
indicating the extent to which the current data element is 
10 potentially decodable at the receiving device. For example, 
the current data element may require information of another 
data element in order to be decodable, and therefore the 
decodability may depend on the availability of the other data 
element at the receiving device. 

15 

In another example the decodability depends on the 
probability of an error free transmission of the currently 
considered data element to the receiving device. This 
probability will generally depend on the characteristics of 

2 0 the transmission channel or network used for transmitting the 

data element from the transmitting device to the receiving 
device, as it will be outlined further below. The error 
probability may be a fixed value or may be a variable entity, 
depending on varying conditions on the transmission medium, 
25 e.g. transmission channel or transmission network. 

The decodability of the currently considered data element can 
be determined on the basis of information on which of a 
plurality of data elements was transmitted to the receiving 

3 0 device. This information may for example be obtained locally 

within the transmitting device, e.g. from a transmitter 
controller such as the transmitter controller 103 shown in 
Fig. 1. In one example the transmitter controller maintains a 
log of all data elements, which have been transmitted to the 
3 5 receiving device, i.e., which have been placed on the 

transmission medium in a 1 send and forget' scheme. In this 
case it may be assumed that the data element placed on the 
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transmission medium actually arrive at the receiving device 
and thus be available at the receiving device. 

In addition thereto assumptions may be made on the basis of 
5 an error probability, i.e. that a data element placed on the 
transmission medium actually arrives uncorrupted at the 
receiving device. The probability may be determined 
heuristically, by measuring the characteristics of the 
transmission channel. For example, in a wireless environment 
10 signal characteristics can be measured to deduce a 

probability of uncorrupted transmittal of a data element to 
the receiving device. In a packet network measurements may be 
made as to the probability of loosing a data packet on its 
way to the receiving device. 

15 

According to another embodiment the information on which of 
the plurality of data elements was already transmitted to the 
receiving device may be obtained on the basis of an 
acknowledgement sent back from the receiving device to the 
2 0 transmitting device, indicating which of the data elements 
were received and error free at the receiving device, or 
which were successfully decodable at the receiving device. 

After determining the decodability of the current data 
2 5 element, in an operation 2 02 a priority is assigned to the 

current data element based on the determined decodability. In 
one example the priority of the current data element is set 
equal or proportional to the previously determined 
decodability of this data element. 

30 

Further, the priority of the current data element may be 
selected on the basis of further information regarding the 
data element, such as its importance for the data stream 
transmission, e.g. decodability of the entire data stream. 
35 For example, a data element may contain basic information 

specifying the characteristics of the data stream including 
partitioning of data elements, transmission rates and 
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similar, and the absence of the availability of this 
information at the receiving device may make it difficult or 
impossible to decode the data stream. 

5 Alternatively or in addition thereto the priority of the data 
element may be set based on information on the importance of 
the information content of the data element for a user 
handling the data stream at the receiving device. For 
example, in a video or audio sequence certain data elements 

10 of the data stream may contain important notifications or 
indications for the user and these data elements will be 
associated with a priority. Likewise, data elements with 
information of lesser importance, such as data elements 
containing purely image data or audio data containing 

15 background noise may be associated with a low priority, as 

their content will neither impair the overall handling of the 
data stream at the receiving device, nor severely degrade the 
quality of the data stream as perceived by a user. 

2 0 The priority value of the data element may be obtained by 

appropriately processing the above additional information on 
the data element with the determined decodability of the data 
element. In one embodiment the priority is determined by 
multiplying the above additional information on data elements 
25 with the previously determined decodability. However, other 
arithmetic operations are conceivable , such as adding the 
additional information and the determined decodability. 

Thereafter, in an operation 203 it is determined whether 
30 another data element is present. If in operation 203 the 

decision is "YES", indicating that another data element is 
present, the flow of operations returns to operation 2 01, to 
determine the decodability of the next data element. In 
practical cases the data stream will comprise a large number 

3 5 of data elements, and according to one example all of the 

data elements of the data stream may be considered together, 
i.e., processed in the loop of operations 201, 202, and 203. 
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According to another example a. predetermined number of data 
elements is processed concurrently in the loop of operations 
201, 202, and 203, i.e., a predetermined amount of data can 
5 be processed concurrently. The data stream may for this 
purpose be arbitrarily divided, into corresponding sets of 
data elements, or the data stream may be partitioned 
according to logic entities, such as data elements of a 
portion of the data stream with mutual dependencies. If 
10 between two groups of data elements no interrelation exists, 
each of the groups of data elements can be processed 
independently . 

According to still another example, the data elements may be 
15 grouped to conform to certain time periods, i.e., in a video 
or audio data stream, data elements can be grouped in 
accordance with a certain playback period of time, such as 
for example one second or any other time period. The groups 
of data elements may be determined on the basis of a buffer 
2 0 size of buffer sizes available at the receiving device, so 
that it can always be assured that a buffer overflow at the 
receiving device is avoided. 

If in operation 203 the decision is "NO 11 , indicating that a 

2 5 next data element is not or presently not available, the flow 

proceeds to operation 204, and. the transmission of the data 
elements to the receiving device based on the priority is 
scheduled, for example by the transmitter controller 103 
shown in Fig. 1. 

30 

The transmitter controller may handle a group of data 
elements concurrently, i.e. may consider one group of data 
elements at a time for a transmission. The transmitter 
controller at first then selects data elements with the 

3 5 highest priority for transmission to the receiving device, 

and thereafter data elements with the second highest priority 
and so on, until either all data elements have been 
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transmitted or the available bandwidth for the data stream 
transmission is exhausted. Data packets of lower priorities, 
which cannot be transmitted within the available bandwidth 
are dropped or discarded. 

5 

According to an alternative operations 201 r 2 02, and 2 04 may 
be executed sequentially for each data packiet, before a next 
data element is considered. In this case, ±n operations 2 04 
each individual data element processed in operations 2 01 and 
10 202 is considered for transmission. 

In a practical case in the transmitter controller such as the 
transmitter controller 103 of the Fig. 1 w±ll decide to 
transmit each data element based on the priority it received 
15 as soon as the element becomes available at the controller. 
Any data element having the highest priority will be 
transmitted immediately, so that highest emphasis is put on 
transmitting data packets with the highest priority. Only in 
case the transmitter, due to sequential processing and 

2 0 transmitting of the data elements, does not have any highest 

priority data element for transmission cueing, the 
transmitter controller will turn to data elements having the 
second highest priority. For this purpose the transmitter 
controller may accumulate a certain number of data elements 
25 associated with priorities, depending on tlie transmission 

bandwidth available for transmitting the data elements to the 
receiving device. 

If the transmission bandwidth is large enough to accommodate 

3 0 all data packets of the data stream, the transmitter 

controller will be able to transmit each data element upon 
receiving it, no matter what priority was assigned to the 
data element. 

35 However, if the transmission bandwidth of the transmission 
medium is lower than required for transmitting all data 
elements of the data stream in a timely manner, the 
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transmitter controller will gradually accumulate data 
elements, as they cannot be transmitted at the same rate as 
they are provided through operations 201 and. 202. It is noted 
that the rate of providing data elements through processing 
5 of operations 201 and 2 02 preferably matches a predetermined 
rate of the stream, i.e., for real time video display or 
audio playback. 

The data elements thus received at the transmitter controller 
10 may be held in a buffer, and the transmitter controller 

attends to transmission of the data elements accumulated in 
the buffer according to their priority. As noted above the 
data elements with the highest priority reaching the buffer 
will consequently be transmitted as soon as possible, which 
15 will be at once, if no further data element having a highest 
priorities available, or which will be aft sir a 11 other 
previously received data elements of highest priority were 
transmitted. Thereafter, the transmitter controller turns to 
presently available data elements of the second highest 

2 0 priority and thereafter to data elements with gradually lower 

priority. Again, if the transmission bandwiclth is not 
sufficient for transmitting all data elements of the data 
stream, data elements with lower priority will gradually be 
accumulated in the buffer, as due to the arrival of data 
25 elements having higher priority, the transmitter controller 
does not attend to transmitting the data elements of the 
lower priorities. In practical cases a certain buffer size 
will be provided at the transmitting device, and in practical 
cases, upon reaching the buffer limit at the transmitting 

3 0 device, data packets with the lowest priorities may be 

discarded from the buffer. 

If after a certain period with low available transmission 
bandwidth the transmission bandwidth is again increased, the 
3 5 transmitter controller will also be able to attend to the 

transmission of data elements with lower priorities and the 
buffer will gradually be emptied. 
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It is noted that in addition to the priority base 
transmission scheme, a time limit may be introduced in 
association with each data element, the time limit indicating 
5 a latest time for transmitting the data element, in o^rder to 
for example maintain real time playback of a data stream. Any 
data elements having exceeded their latest transmission time 
may then be discarded, as known in the art. 

10 In the following a further embodiment of the invention will 
be described with regard to Fig. 3. 

Fig. 3 illustrates operations of a method for prioritising 
data elements of a data stream for transmission to a 
15 receiving device according to another embodiment of tine 

invention. Fig. 3 particularly illustrates operations for 
determining the decodability of data elements and the 
operations of Fig. 3 may be carried out using the hardware 
arrangement of Fig. 1, however, Fig. 3 is not limited 

2 0 thereto. 

In a first operation 3 01 at the transmitting device data 
elements of the data stream are appropriately grouped. The 
grouping may be as it was outlined with regard to Fig. 2, 
25 i.e. predetermined or variable numbers of data elements may 
be grouped, or a grouping of data elements according to a 
playback period of time at the receiving device may be 
carried out. Also, the entire data stream may be considered 
as one group . 

30 

Thereafter, in an operation 3 02 it is determined whether a 
currently considered data element requires a data element for 
being decodable at the receiving device, i.e. whether- it 
requires a reference data element . Depending on the used 

3 5 codec, there may be interdependencies between data elements , 

i.e., for the decoding process of one data element another 
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data element may be required. For example, in a predictive 
coding scheme such data element dependencies are present. 

According to an example, in a video sequence of images, a. 
data element may contain image information, for example, of 
pixels making up one frame of the video sequence. 
Alternatively, in a sequence of images making up the video 
stream, a data element may contain difference information, 
between a previous frame and a current frame. The current 
frame can then be calculated by combining the different 
information and the image information of the previous data 
element. Accordingly, at the receiving device the previous 
data element is required in order to appropriately obtain, or 
decode the current data element. Moreover, data elements are 
conceivable, which simply contain information indicating that 
there was no change between the previous data element and the 
current data element. Accordingly, dependencies may exist 
from a current data element to a previous data element or 
elements, the previous data element (s) thus being required at 
the receiving device for decoding the current data element. 

In operation 3 02 the currently considered data element may be 
analysed to determine whether a dependency to a previous data 
element exists. This information may be obtainable from the 
data element itself, or may be derivable from a data element 
type. Further, in operation 3 02 the identity of the required 
reference data element it is determined. 

In an operation 3 03 it is decided, whether a reference data 
element is required. If in operation 303 the decision is 
"YES 11 , indicating that a reference data element is required, 
in an operation 3 04 it is determined whether the reference 
data element has been transmitted. As noted before, the 
transmission of the reference data element can be detected, 
locally at the transmitting device, i.e. it can be checked, 
whether the reference data element was placed on the 
transmission medium, or, acknowledgement information from the 
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receiving device may be used, indicating whether the data 
element was received and/or successfully and without errors 
decoded at the receiving device. 

5 In operation 3 05 it is then decided whether the reference 
data element has been transmitted. If in operation 3 05 the 
decision is "YES", the flow continues with operation 306, in 
which case the decodability of the current data element is 
set equal to the decodability of the reference data element. 

10 This reflects the fact, that the current data element is at 
best only decodable to an extent of the decodability of the 
reference data element. For example, if the reference data 
element is only partially decodable, the partial decodability 
will also affect the decodability of the current data 

15 element . 

According to an example, if the decodability of the reference 
data element is one, indicating that the reference data 
element is fully decodable, the decodability of the current 

20 data element will also be set to 1, as in this case the 

current data element will also be fully decodable. Likewise, 
if the decodability of the reference data element is 0, i.e. 
that the reference data element is not decodable, the 
decodability of the current data element will also be set to 

25 0 . 

The flow then continuous to operation 307, where the priority 
of the current data element is set based on the decodability 
in operation 306. Operation 307 may be as outlined with 
3 0 regard to previous embodiments. 

Thereafter, in an operation 3 08 it is determined whether a 
next data element of the group of data elements built in 
operation 301 is present. If in operation 3 08 the decision is 
35 "YES", indicating that a further data element is present, the 
flow of operations returns to operation 3 02 for considering 
the next data element. If in operation 308 the decision if 
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"NO", indicating that a next data element of the group of 
data elements is not available, in an operation 3 09 the 
transmissions of data elements to the receiving device is 
scheduled based on the respective priorities. Thereafter, the 
5 flow of operations may continue to a further group of data 
elements or if the end of the data stream is reached, 
operations may terminate. 

If in operation 3 03 the decision was "NO", indicating that a 
10 reference data element was not required, tine flow of 

operations may directly continue to operation 308, i.e., to 
the next data element, if available. 

Further, if an operation 305 the decision -was "NO", 
15 indicating that the reference data element was not 

transmitted, operations 3 06 and 3 07 are skipped, i.e. the 
decodability of the current data element a:nd the priority 
setting is not performed, i.e. the decodability of the 
current data element remains as before. It is noted that 
20 initially, i.e., prior to first consideration of each 

particular data element the decodability will preferably be 
set to 0 . 

In an alternative, a current data element may not only depend 

2 5 on one reference data element, but may depend on a plurality 

of such reference data elements. In this case the 
decodability of the current data element may be set based on 
the decodabilities of the group of reference data elements. 
For example, the decodability of the current data element is 

3 0 set to an average decodability of the group of reference data 

elements. Alternatively, the decodability of the current data 
element may be set to the decodability of "the reference data 
element having the lowest decodability. 

3 5 In the following a further embodiment of t Hie invention will 
be described with regard to Fig. 4. 
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Fig. 4 illustrates operations of a method for prioritising 
data elements of a data stream for transmission to a 
receiving device and particularly illustrates operations to 
recalculate a decodability of at least a portion of the data 
5 elements of the data stream upon transmission of the data 
elements. The operations of Fig. 4 may be carried out using 
the hardware arrangement of Fig. 1, however, Fig. 4 is not 
limited thereto. Fig. 4 illustrates a continuous updating of 
the decodabilities of the not yet transmitted data elements 
10 on the basis of information regarding data elements that have 
been transmitted. 



In a first operation 4 01 it is determined that a current data 
element has been transmitted. For example, the determination 

15 may result from a transmittal report from a transmitter 

controller such as the transmitter controller 103 of Fig. 1, 
as noted before. Further, the information on the transmission 
of a data element could be obtained from the receiving 
device, reporting reception and/or error- free decoding of the 

20 current data elements. 



In a transmission scheme with a compression of the stream 
information into data elements that may have dependencies on 
one another, successful decoding of a data element at the 

2 5 receiving device may depend on the presence of information 

from another data element at the receiving device. Only if 
this information from the reference element is available, the 
dependent data element can be decoded. 

3 0 Therefore, in an operation 4 02 all data elements requiring a 

transmission of the current data element for decoding are 
determined. The information on which data elements depend 
from the current data element, i.e., require information from 
the current data element for successful decoding at the 
3 5 receiving device, may be obtained by analysing the respective 
data elements, for example, information on dependent data 
elements could be noted in the current data element, or 
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likewise, a reference data element, i.e., the current data 
element, could be identified in the dependent data elements. 
Still further, information on data elements depending from 
another data element may be obtainable from the position of a 
5 data element in a sequence of data elements or by a type of 
data elements. For example, in a decoding scheme dependent 
data elements may be identified as dependent and may always 
depend on a preceding data element . 

10 According to another example, decodability determining means 
such as the decodability determining means 101 of Fig. 1 
reviews a dependency record of the current data element, 
wherein the dependency record indicates all data elements 
requiring the current data element for decoding. The 

15 dependency record of the current data element may be stored 
in association with the current data element or may be 
available from a data stream log or similar sources. 

Upon determining all data elements depending on the current 
2 0 data element, in an operation 4 03 the decodability of the 
determined dependent data element is recalculated. The 
calculation of the decodability may be as outlined with 
regard to previous embodiment . 

2 5 In an example, further to the above, in operation 4 03 the 

decodabilities of all data elements are updated which depend 
from anyone of the data elements dependent from the currently 
considered data element and having recalculated decodability. 

3 0 Thereafter, the flow may continue with detecting another data 

element transmission in operation 4 01, or, upon reaching the 
last data element of the stream, the flow may end. 

The embodiment of Fig. 4 thus allows reducing a processing 
3 5 requirement for updating the decodabilities of the data 

elements upon transmitting a data element. Only data elements 
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depending from the transmitted data element are updated 
regarding their decodability . 

Based on the updated decodabilities of the determined data 
5 elements, the priorities may also be recalculated, such as by 
prioritising means 102 shown in Fig. 1, and the transmitter 
controller may proceed to schedule the transmission of the 
updated data elements according to their new priorities, as 
outlined before. 

10 

The operations of Fig. 4 may generally be carried out at any 
point in time after a data element was transmitted. The 
updating procedure may also be carried out after a certain 
number of data elements have been transmitted. For example, 
15 the updating operations of Fig. 4 may be carried out 

following operation 2 04 of Fig. 2, or operation 3 09 of Fig. 
In the following a further embodiment of the invention will 
be described with regard to Fig. 5. 

2 0 Fig. 5 illustrates operations of a method for prioritising 
data elements of a data stream for transmission to a 
receiving device according to another embodiment of the 
invention, particularly outlining operations to improve a 
prioritisation of the data elements. The operations of Fig. 5 

2 5 may be carried out using the hardware structures shown in 

Fig. 1, however, Fig. 5 is not limited thereto. 

It was outlined before that a priority of the data elements 
can be determined on the basis of the decodability of each 

3 0 respective element, and the priority assignment may further 

include information on the type or nature of the data 
element, see the above embodiments. To further improve the 
prioritisation of a data element, the priority of a data 
element in the present embodiment is further selected on the 
3 5 basis of an overall improvement of the data stream achievable 
by transmitting the data element under consideration. 
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In a first operation 501 an average decodability of a number 
of data elements is determined. The average decodability may 
be an arithmetic mean value of decodability of the data 
elements, i.e., a sum of all decodabilities of all data 
5 elements divided by the number of data elements. However, in 
an alternative, other decodability values reflecting the 
decodability of a number of data elements under consideration 
may be used. The number of data elements may correspond to a 
predetermined amount of data, i.e., a predetermined number of 

10 data elements may be grouped. Preferably, as noted before, 
the group of data elements is selected such that 
interrelations with other groups of data elements do not 
exist. Alternatively, however, it is also possible that a 
grouping of data elements according to playback times at the 

15 receiving device are used, or that the entire data stream is 
considered simultaneously. 

The average decodability of the number of data elements may 
for example be calculated by decodability determining means 

2 0 such as the decodability determining means 101 shown in Fig. 

Then, in an operation 5 02 it is assumed that a currently 
considered data element is transmitted and the effect of the 
25 transmission of the currently considered data element on the 
average decodability of the number of data elements is 
determined. Thus, in operation 5 02 an increase of the average 
decodability by transmitting the current data element is 
determined . 

30 

For example, if in a sequence of data elements each 
individual data element depends from the previous data 
element in the sequence, i.e., if each data element requires 
information from a preceding data element for being decodable 

3 5 at the receiving device, and only the first data element of 

the sequence of data elements does not depend on another data 
element, the decodability of each data element except the 
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first one is zero, as long as the information of the first 
data element is not available at the receiving device. Thus, 
a transmission of the first data element will lead to a large 
increase of the average decodability of all considered data 
5 elements, as only with the first data element the subsequent 
data elements can be decoded at the receiving device. Any 
other data element in the data element sequence of this 
• example will have a lesser effect, namely only onto its 
following data elements. 

10 

The above example illustrates that depending on the 
interrelation between the data elements of the data stream 
the transmission of a particular data element may have a 
larger or smaller effect on the overall decodability of the 
15 data stream. 

The average decodability increase of the group of data 
elements under consideration then is used in an operation 503 
for re -determining the priority of the current data element 
2 0 based on the present decodability of the current data element 
and the decodability increase of the number of data elements. 
For example, the priority of the data element under 
consideration could be multiplied by a value representing the 
average decodability increase, e.g. a normalised average 

2 5 decodability increase value or based on another arithmetic 

operation . 

The operations of Fig. 5 may be carried out in association 
with the prioritising operations illustrated with regard to 

3 0 previous embodiments, i.e. operations 2 02 of Fig. 2 and 3 07 

of Fig. 3, or at any other suitable point in time. 

According to an alternative, the decodability increase, i.e., 
the increase of the average decodability of the number of 
3 5 data elements under considerations may be set into 

perspective to the size of the current data block, i.e. the 
bandwidth requirement for transmitting the data element. 
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Accordingly, the decodability increase determined in 
operation 5 02 achievable by transmitting the data element 
under consideration may be divided by the size of the 
currently considered element. Thus, the bandwidth 
5 requirements for transmitting the data element may also be 
entered into the process of determining a priority. 

Accordingly, the embodiment of Fig. 5 can be used to 
differentiate between important and unimportant packets so 

10 that less important packets can be skipped if there is not 
enough bandwidth available. When the skipping of lesser 
important data elements will take place via the operations 
outline with regard to previous embodiments in view of 
scheduling the transmission of data elements according to 

15 their priority. 

In the following further embodiment of the invention will be 
described with regard to Fig. 6. 

20 Fig. 6 illustrates operations of a method for prioritising 
data packets for transmission to a receiving device, 
particularly outlining the handling of data elements having 
different types. The operations of Fig. 6 may be carried out 
using the hardware structures shown in Fig. 1, however, Fig. 

25 6 is not limited thereto. 

In Fig. 6 it is assumed that a data stream comprises 
independent data elements being independent from other data 
elements in that they can be decoded at the receiving device 

3 0 without requirement of information from other data elements. 
Further, the data stream is assumed to comprise dependent 
data elements, being data elements, which are at least one 
other data element or reference data element. As noted 
before, a dependent data element requires information from at 

3 5 least one another data element for being decodable at the 
receiving device. 
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In a first operation 601 the data element type is determined, 
i.e., it is determined whether the data element is an 
independent data element or a dependent data element . The 
type may be determined by analysing the data element, or by 
5 reading an identifier of the element, the identifier 
indicating its type. 

Thereafter, in an operation 602 it is decided whether the 
data element is an independent data element or a dependent 

10 data element. If an operation 602 the decision is "YES", 
indicating that the data element is an independent data 
element, in an operation 603 the decodability is set to a 
maximum value. The maximum decodability reflects the fact 
that the independent data element is fully decodable at the 

15 receiving device. Thereafter, the flow continues with the 
next data element and operation 601. 

If in operation 602 the decision is "NO", in an operation 604 
the data element is identified as a dependent data element. 
2 0 Thereafter, in an operation 605 the decodability of the 

dependent data element is set based on the decodability of 
the reference data element or data elements. If the data 
element is dependent only on one other data element, the 
decodability of the currently considered dependent data 

2 5 element is set to be equal to the decodability of the 

reference data element. It is noted that the reference data 
element itself may be a dependent data element, depending 
from yet another data element, and its decodability being set 
to be equal to the decodability of the further reference data 

3 0 element. 

In operation 605, if the decodability of the data element 
under consideration is set, all data elements itself 
depending on the currently considered data element have to be 
3 5 updated. Thus, the decodability of all data elements 

depending from the currently considered (dependent) data 



WO 2005/076218 



35 



PCT/EP2004/000837 



element will correspondingly be updated, i.e., set to the 
decodability of the respective reference data elements. 

If the data element under consideration depends from a 
5 plurality of reference data elements, the decodability may be 
set based on the decodability of the number of reference data 
elements. For example, the decodability may be set to an 
average decodability of all reference data elements, or to 
any other value based on the decodabilities as noted before. 

10 

Thereafter, the flow returns to operation 601, in order to 
process the next data element. 

The embodiment of Fig. 6 illustrates the dynamic behaviour of 
15 the decodabilities associated with each data element, based 
on dependencies between data elements, and based on the fact 
that reference data block has been transmitted. 

In the following a further embodiment of the invention will 

2 0 be described with regard to Fig. 7. The operations of Fig. 7 

may be carried out using the hardware arrangement of Fig. 1, 
however, Fig. 7 is not limited thereto. 

Fig. 7 further illustrates operations for considering an 
25 error rate of a transmission channel between the transmitting 
device and the receiving device. For example, a transmission 
may be error-prone, or data elements will be lost during 
transmission, e.g. due to a congestion condition in a 
computer network. Especially in wireless connections 

3 0 transmission errors are frequent, and thus it may be 

advantageous to use information on the error rate of the 
transmission medium. 

The embodiment of Fig. 7 is particularly advantageous in case 
3 5 information from the receiving device regarding successfully 
received and/or decoded, i.e. error free, data elements is 
not available. In the embodiment of Fig. 7 the transmitting 
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device makes assumptions regarding the likelihood of 
successfully transmitting and decoding data elements at the 
receiving device, and uses this information for calculating 
or recalculating decodabilities and/or priorities. 

5 

In a first operation 701 the probability of an error free 
transmission over the used transmission medium is determined. 
The probability of an error free transmission may be 
determined heuristically, i.e. measurements may be made based 

10 on transmission on test data elements, measurements of signal 
characteristics and similar. Further, a probability of an 
error free transmission may depend on and thus be derived 
from information available regarding the transmission medium, 
e.g. a traffic load in a network, terrain conditions in 

1 5 wi re less envi r onment s and s imi 1 ar . 

The probability of an error- free transmission may be 
determined once or be determined repeatedly, e.g. if varying 
transmission conditions are present. Thus, the error 
2 0 probability may be dynamically adapted to the actual 
condition of the transmission medium. 

Then, in an operation 702 a data element type is determined, 
i.e., it is determined whether the data element is an 
2 5 independent data element or a dependent data element, similar 
to operation 601 of Fig. 6. 

In an operation 7 03 it is decided whether the data element is 
an independent data element, and if in operation 7 03 the 

30 decision is "YES", indicating that the data element is an 
independent data element, in an operation 704 the 
decodability of the data element under consideration is set 
equal to the probability of an error-free transmission. For 
example, if the probability of an error-free transmission of 

35 the data element to the receiving device is 0.99, the 

decodability of this data element may be set to 0.99, or to 
any other value proportional to 0.99. 
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If in operation 703 the decision is "NO", in operation 705 it 
is known that the data element is a dependent data element, 
.i.e., a data element being dependent from one or more 
5 reference data elements. 

Thereafter, in an operation 706 the decodability of the data 
element under consideration is set equal to the decodability 
of the reference data element multiplied by the probability 
10 of an error- free transmission. 

Similarly, if a plurality of reference data elements is 
present, the decodability obtained on the basis of the 
decodabilities of the reference data elements may be 
15 transmitted by the probability of an error-free transmission 
of all reference data elements. For example, if two reference 
data elements are required at the receiving device for 
decoding the currently considered data element, and if a 
probability of an error-free transmission is 0.99, the 

2 0 probability of error-free transmission of both data elements 

is 0.99 x 0.99, i.e. 0.9801. 

After operations 704 and 706 the flow continues with 
operation 702, i.e., the next data element is considered. 

25 

As shown with the above operations, the embodiment 
illustrated with regard to Fig. 7 allows to also consider the 
conditions of the transmission medium for determining a 
decodability and priorities of the data elements. 

30 

In the following a further embodiment of the invention will 
be described with regard to Fig. 8. 

Fig. 8 illustrates operations of a method for prioritising 

3 5 data elements for transmission to a receiving device, 

particularly outlining an example of a video stream. The 
operations of Fig. 8 may be carried out using the hardware 
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arrangement of Fig. 1, however, Fig. 8 is not limited 
thereto . 

In Fig. 8 it is assumed that a data-encoding scheme is used 
5 which provides three different types of data elements of the 
video stream, in the present embodiment called blocks . A 
first type of blocks, so-called intra-blocks , contains 
complete information of that block and can therefore be 
independently decoded at the receiving device. A second type 

10 of blocks, so-called inter-blocks encode the differences 
between the actual content of the block and previous 
contents. Further, an inter-block may contain a motion vector 
for describing the displacement of the content, e.g. picture 
elements, between the previous content and the present block. 

15 For decoding an inter-block, previous content is necessary. A 
third type of blocks, so-called skip-blocks indicate that at 
least a part of the skip-block is identical to at least part 
of a previous block and therefore the skip-block only 
contains a reference to that previous block in the data 

2 0 stream. For decoding a skip-block at the receiving device, 
the previous content is therefore necessary. 



Intra -blocks , inter-blocks and skip-blocks may be transmitted 
in groups or interleaved with one another, is required and 
2 5 several intra-blocks , inter-blocks and skip-blocks may make 
up a macro-block or frame of the data stream. 

In the following the operations of the current embodiment 
will be outlined in further detail . 

30 

In a first operation 801 the probability of an error-free 
transmission of a currently considered data block over the 
transmission channel is estimated or a previous estimation 
value for a probability of an error- free transmission is 
35 used. The estimation of operation 801 may be as described 
with regard to operation 701 of Fig. 7. 
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In an operation 8 02 the data block type of the currently 
considered data block is determined, i.e., it is determined 
whether the data block is an intra-block, inter-block or 
skip-block. The determination operations may be similar to 
5 operation 702 described with regard to Fig. 7. 

In an operation 8 03 it is then determined whether the data 
block is an intra-block . If an operation 8 03 the decision is 
"YES", in an operation 804 the decodability of the intra- 
10 block is set equal to the probability of an error-free 
transmission. 

If in operation 803 the decision is "NO", indicating that the 
data block is not an intra-block, it is determined in 

15 operation 805 whether the data block is an inter-block. If 
the decision is "YES", indicating that an inter-block is 
present, in an operation 806 the decodability of the inter- 
block is set equal to the decodability of the reference data 
block multiplied by the probability of the error-free 

2 0 transmission over the transmission channel. 

Similarly, if the inter-block depends from a number of 
reference blocks, the decodability of the number of reference 
blocks, the decodability of the inter-block may be set based 
25 on the decodabilities of the reference blocks, as outlined 
before, multiplied by the probability of an error-free 
transmission of all reference data blocks, as outlined 
before . 

30 If in operation 805 the decision is "NO", in an operation 807 
it is declared that the data block is a skip-block, and in an 
operation 808 the decodability of the skip-block is set equal 
to the decodability of the reference data block or blocks, 
independent of the probability of an error-free transmission. 

35 This reflects the fact that the skip-block itself does not 
contain any image information but only refers to previously 
transmitted content . 
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According to another alternative, to reduce processing and 
transmission requirements, the decodability determining means 
may be adapted to set the decodability of the reference data 
5 element equal to the decodability of a data frame containing 
a multiple data elements including the reference data 
element. Thus, the decodabilities may only be determined on a 
frame basis and processing requirements may be reduced. 

10 In a video streaming application motion compensation and thus 
motion vectors are normally considered for determining a 
decodability and/or priority. For example, a motion vector 
leads to a dependency of a data element from another data 
element and thus has influence on the decodability. However, 

15 for reducing processing requirements in a video streaming 

application, in another embodiment motion compensation in a 
video stream is disregarded. In an example, motion 
compensation/estimation is disregarded in connection with 
determining a decodability and thus in connection with 

2 0 assigning priorities. Accordingly, data elements, as 

described in the previous embodiments, or data blocks in the 
present case may encode differences of present image 
information of a data element to image information of a 
previous data element, but motion vectors, i.e. motion 
25 estimation and a motion compensation, are disregarded. 

Without handling motion vectors, decodability determining 
operations, prioritising operations may be significantly 
reduced and a complexity reduction may be achieved. 

3 0 Even though in the present embodiment a video stream is 

considered, any other kind of data stream may be 
correspondingly processed, for example an audio stream or any 
other kind of data stream subdivided into data blocks 
corresponding to intra -blocks , inter-blocks and skip-blocks. 

35 

In the following a further exemplary embodiment of the 
invention for decodability based priority assignment for data 
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streaming is described. In the this embodiment it is assumed 
that the data stream is a video stream, however, the 
embocdiment is not limited thereto. Further, it is noted that 
all elements of the embodiment are all viewed as optional and 
5 advantageous features of the invention. 

■ * 

As noted before, streaming data such as video over packet - 
switched networks is an application of growing significance. 
Due to the best effort property of the Internet, it may not 

10 be possible to transmit the whole data stream to the 

receiver. Especially in the case of wireless streaming the 
available bandwidth is highly variable. These bandwidth 
limitations in combination with the real time constraints 
related to streaming result in protocol based packet drops at 

15 the server, because the data can not be delivered timely. 
Yet, there was no effective solution to avoid such 
situations, but their effects can be mitigated by different 
means . 

2 0 Video decoders support error concealment algorithms, and are 
able to recover much of the encoded video information even if 
some parts of the encoded stream have not been transmitted. 
The effectiveness of these algorithms depends on the 
scalability support of the codec and the position of the 

2 5 error within the stream. This paper proposes a new method to 

scale the bandwidth requirement of a H.2 63 or MPEG video 
stream without relying on support of the codec. Its basic 
idea is to use the available bandwidth for the transmission 
of the more important parts of the encoded video stream. 

3 0 Temporal scalability is simulated by dropping several frames 

of the non- layered video stream. If the dropped frames are 
selected randomly, heavy distortions will arise. The first 
and simplest step to improve the presentation quality is 
assigning different priority values to the different frames 
35 with±n the single layer, and drop less important frames 

first . These values indicate the importance of a frame for 
the presentation of the video stream, and can either be 
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For block based video codecs, the obvious concept is to 
derive the priorities from the different frame-types (I,P,B). 
5 An I -Frame is Intra-encoded and can be decoded on its own. P- 
Frames depend on an I- or other P-Frames and may contain only 
differences to these frames. B-Frames are a second type of 
dependant frames . 

10 The present example describes to quantify the information 

available in the data frames of a video stream based on the 
decodability of the video stream. The decodability measures 
which fraction of a video is expected to be decodable at the 
client. The proposed example can be used similarly to the 

15 rate-distortion optimized algorithms for a dynamic generation 
of priorities, but does not rely on the unknown distortion 
values . It is based on the internal structure of the stream, 
which can either be derived from the encoder logs or 
extracted from the stream. 

20 

The example can be generalized to other data sets than video 
streams containing dependent, independent and transparent 
elements . 

2 5 In the present embodiment it is assumed that the basic data 
unit of a transmission system is a frame and these logical 
frames are considered to be the same as the encoded video 
frames, i.e. data elements. The model of the video stream is 
e.g. based on the H.263 video codec. B-Frames and Reference 

30 Picture Selection are neglected within this simplified model, 
so the GOFs consist of I -Frames with consecutive P-Frames 
depending on the previous frame only. If one frame within a 
dependency chain is missing, the decoding of consecutive 
frames can not be guaranteed, and they might be erroneous. 

35 

E sich. frame is divided into so-called macro-blocks. Each 
macro-block can be of the type Intra, Inter or Skip. Intra- 
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Blocks contain the complete information of that block and can 
be decoded independently. Inter-Blocks encode the differences 
between the actual content of the block and previous content. 
Motion compensation describes the displacement of the content 
5 between the frames. Motion compensation does not influence 
the decodability measure. Skip-Blocks indicate that the 
content of an area represented by the block exists in the 
previous frame, too. 

10 It is assumed that one frame consists of one packet only. 
Given that each of the packets contains synchronization 
information, and the information within a packet is not lost 
if previous packets of the same frame are lost, the frame 
error rate and the packet error rate are the same. Having 

15 dependencies between packets, the frame error rate has to be 
derived from the packet error rate . 

The basic idea is to optimize the number of decodable frames 
under the assumption that each decoded frame has got the same 
2 0 importance. To achieve that goal, additional information 

about the structures and dependencies of the encoded video 
frames is used to calculate the priority. 

Decoding errors within one frame may propagate to subsequent 

2 5 video frames because of these dependencies, but these errors 

may be corrected by independently decodable Intra -Blocks 
late r . 

Each frame k: consists of several macro-blocks. The 

3 0 decodability Ik,i of the macro-block 1 represents the 

probability by which this block can be decoded without 
errors. These Ik,i depend on the expected error rate Ask of 
the encoded frame, the macro block type (Inter-, Intra- or 
Skip) , eventually on a block Ik-i,i' of the previous frame, and 
3 5 can be written as 
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(1.) 
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blocks 



The reference block Zfc-i,i' need not be a macro block, but can 
be any 16x16 block of the previous frame (given that half - 
5 pixel motion compensation is neglected) . Skip-blocks simply 
copy the contents of the previous frame, therefore Ik-i,i>= Ik- 
1,1 . 

The expected error rate Sk is not only influenced by 
10 transmission losses, but by the transmission algorithm as 
well. The decision not to transmit a packet leaves £k-1.0. 
The expected error rate of a frame being transmitted is be 
the error rate of the transmission path. 



15 Fig. 9 illustrates the different values of Sk given that no 

forward error correction (FEC) is used in a state diagram of 
the expected error Sk. 

A subset Ni, k of the macro-blocks belonging to a frame are 
2 0 independently decodable Intra-blocks , Ns,k shall contain all 
Skip-blocks, and finally Np,* represents the Inter-blocks. 
Using this definition, the decodability Ik of a frame can be 
defined as the mean value of its block decodabilities 



25 



(2 . ) 
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where ru,k np,k and ns f k are the size of the sets Nx.k Np,k and Ns,k 
respectively, and n= m^+np^+ns^. The more important symbols 
are subsumed in the following table . 
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Frame Index 
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Index of Frame to 
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Decodability of a frame 
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Ask* 


Expected change of £k' due 
to transmission 


n 


Number of blocks per 
frame 


Ill,k 


Number of Intra- Blocks in 
frame 


IlP,k 


Number of Inter-Blocks in 
frame 


TlS,k 


Number of Skip-Blocks in 
frame 




prioricy or rrarcie k 




Effect of the 
transmission of frame k' 

on frame k 


1 Pk \ 


Normalized priority of 
frame k 


Sk 


Size of frame k 



10 



If there are enough Inter- and Skip-Blocks, and relations in 
the space or time dimension are neglected, the decodability 
of the blocks of the previous frame can be approximated by 
the total decodability of the previous frame. 



Th.i s f i 1 ters the effect of the motion compensation in the 
frames and simplifies the model. The expression for the 
decodability Ik can then be written as 
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(4.) 



/^(l-,J + I[(w /c ). 



n 




n n 



This recursion can not be written in a closed form without 
further constraints. It can either be applied to an actual 
5 video stream or transformed assuming constant parameters. 

Generally, if Ik-0 / no part of a frame can be decoded; if 
Ik=l, it is guaranteed, that the frame can be decoded. If 
either only a part of the frame can be decoded or there are 
10 probabilities of packet loss, Ik is between these values. 

More interesting than the decodability of the frame is its 
total effect on the decodability of the video stream. The 
information within every macro-block propagates to later 
15 frames due to consecutive Skip- or Inter-Blocks. If there are 
K frames, the decodab±lity of the complete video stream is 



2 0 The transmission of a frame k ' changes its expected error 

rate Sk' . To assess the effect of the transmission of a frame 
on the stream decodab±lity , its partial derivative is needed 



(6.) 



di _ l ^ di /c 
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with 




/cWc 



n 



(7.) 



ds k . 




de k , 



|/c'= k 



30 



This formula illustrates the fact that the information of a 
frame is encoded with±n its Intra- and Inter-Blocks (k'=k) 
and this information propagates through Inter- and Skip- 
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Blocks. There is no back-propagation within the model, so a 
frame can not have any influence on previous ones (the last 
term is always zero for k<k') . 



10 



The transmission of a frame k' has got an effect onto the 
stream decodability. This effect shall be reflected by the 
priority of a frame. If the expected error rate Sk> changes 
from a previous value of e to e-Ask> , the priority Pk> can be 
calculated as 



(8.)P,= JJ^,= f± 



1 ^ . • 



^ a-=i 



with 



15 (9.) J|^,= 



ds k , 



The value of Aa-.> depends on e. Usually, Ask* has a value of 1 - 
FRER, where FRER is the expected frame error rate and depends 
on the network. 

20 

Fig. 10 shows a state diagram of the change Ask> of the 
expected error, no FEC . 

To simplify the notation, a new indicator variable Tk,k*(Ask<) 
25 is defined representing the effect a transmission of frame k' 
has on the decodability of frame k, if the expected error 
rate of k' is reduced by Ask* : 

e-Ae k . qj. 



fir/- 

(10.) 



/z 
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The video stream model does not include back-propagation. 
Therefore, the calculation of ttie priority can be simplified 
by taking only the transmitted and consecutive frames into 
account. Together with the definition of Ik,k* (Ask') , the 
5 priority can be written as 

(11 . ) ^=^Z4,'(^)==^i:4 ft '( A ^) • 

In addition, the valid elements within the sum are all the 
10 consecutive frames within a GOF only. With the beginning of 
the next GOF each Ik,k> (Ask<) will be zero, because np,k=ns,k=0 
for the next I-frame. A scheduler should transmit the frames 
with the highest priorities. 

15 Using the above formulas it is possible to optimize the 

overall quality of a video if only a certain number of frames 
can be transmitted. However, usually not the number of frames 
is the limiting factor but the available bandwidth. This 
results in a limited number of bytes which can be transmitted 

2 0 in a certain time. Therefore, a transmission algorithm has to 
maximize the amount of priority it can transmit in a fixed 
time interval. This can be done by defining and using the 
normalized priority | Pk* | . Assuming a constant bandwidth for 
this time interval, the normalization can be done by 

25 calculating the priority per Byte. Given that Sk> is the 
number of bytes of the encoded frame k ! , its normalized 
priority is 

(12.) 

30 

The necessity of this size correction factor will be shown by 
the simulation results. Approaches with this size factor are 
called ra.te-decodability optimized, or shortly rate optimized. 
If the priorities Pk> are used, the algorithm is frame 
35 optimized. 
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The concept of decodability can be evaluated in several ways 



making different assumptions. Three significant algorithms 
applying the decodability and will be discussed together with 
their assumptions within this section. Two parameters are 
5 varied: The values of of nx,k, ns r k and rip,k are either 

considered constant or taken from the stream, and the frame 
error rates can be assumed constant or calculated 
dynamically. This results in static or dynamic priority 
values during the transmission. 

10 

Static Priorities from Constant Values 

The recurrence for Ik and Tk f k* can be solved and evaluated for 
constant values of ni,k, ns,k, np,* and Sk. Using these fixed 
15 values, static priorities can be calculated. 

Preconditioned that all the variables ni,k, ns,k, xiP t k and Sk do 
not depend on the frame index Vz t the recursions can be 
written in a closed form with the help of the z - 

2 0 transformation. The boundary conditions are given by the fact 
that the first frame is always an I-Frame. The formulas are 
valid for one GOF only, because any consecutive I -Frame has 
m,k=n and would violate the preconditions. However, the GOFs 
can be calculated independently if the index of the I -Frame 

25 is set to zero. The resulting values don't take a certain 
stream into account . 

The decodability Ik of a frame can be written as 



30 



(13 . ) 




The change of the decodability depends on the block error 
rate of the network Ask= 1 - Sk- 1 - FRER . 
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Fig. 10 depicts Ik for several parameter sets with £k= 0.2. For 
the decodability converges to 



i.e. the Intra-Blocks continuously refresh parts of the frame 
and keep the decodability greater than zero, first 1 line 

represents frames where every block type is present (1) . 
Obviously, given that there are I -Frames only, a constant 

10 decodability without any dependency is observed (2) . The same 
is true for P-Frames consisting of Skip-Blocks only, but in 
this case the decodability depends on the first I -Frame (4) . 
The other curves show a decreasing decodability over the 
length of a GOF - either converging to a constant value if I- 

15 Blocks are present (1,3) , or to zero if the number of I - 
Blocks can be neglected (5) . 

Fig. 11 shows a decodability of frames using constant 
parameters GOF of length 15 and Sk- 0.1. 

20 

More important than the decodability is the priority of the 
frames. As shown within the appendix, the dependency between 
the frames is taken into account by 
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(15 . ) 





This formula can be inserted into the expression used to 
calculate the priority Pk* 



30 
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and depicted for different parameter sets again, as it has 
been done in Fig. 12. 



35 



Fig. 12 shows priority of frames using constant parameters 
GOF of length 15 and Sk= 0.1. 
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Fig. 12 reveals the dependencies of the frames. Especially in 
the case of Skip-Blocks, the I -Frame is highly important 
because it is responsible for the presentation of the 
5 following frames (4) . The case of independent frames (2) is 
now distinguishable from the former case, and has a constant 
priority P= (1 -s) . Especially the mixed case (1) is 
noteworthy: The priority remains almost constant after the I - 
Frame and drops towards the end of the OOF , because the 
10 effect on consecutive frames due to Inter- and Skip-Blocks 
obviously ends with the end of the GOF . 

The other cases (3, 5) confirm a decreasing priority towards 
the end of the GOF. The exact type of this decrease depends 
15 on the parameters, but the analytical results support the 
strategy of dropping frames at the end of a GOF. 

Static Priorities from Stream Statistics 

The real block statistics (nl 7 k, nS,k and nP,k) are extracted 
2 0 from the stream or read from the logfile of the encoder; Sk 
is set to a fixed value, leading to static priority values, 
too . 

If the true values of ni,k, ns,k and np,k, are extracted from 

2 5 the video stream and the priorities are calculated using 

these values, the simulation results should improve. However, 
the proper calculation of Sk turns out to be difficult. The 
values of sk reflect two properties of the transmission 
system: The first one is the decision of the scheduler to 

3 0 transmit or skip a packet, the second one is the mean error 

rate of the network. The later effect sliall be neglected for 
simplicity. 

8k shall be approximated by a constant parameter for this 
35 method. However, there are several ways to do so: If the 
skipped packets are uniformly distributed, this constant 
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value should reflect the probability of a skip, i.e. be 
equivalent to the expected fraction of skipped frames. 
However, initial experiments support the results of the last 
section, and frames at the end of a. GOF are skipped 
5 preferentially. To this end the error Sk within the 

dependency chains of the transmitted frames is zero. This can 
be generalized by setting the constant error rate sk to the 
frame error rate of the network. 

10 Nevertheless, the simulation results will show that this 
method did not improve the transmission as expected. 

Dynamic Priorities 

Again, the real block statistics are used, and 8k is updated 
15 while the transmission is in progress. This results in 
dynamic priorities . 

It is possible to apply the formulas without simplifications. 
In that case the varying values of the frame error Sk have to 
20 be taken into account and the priorities will be dynamic. 
Therefore, this priority assignment algorithm is quite 
complex and has an increased processing power requirement. 

The exact values of ni,k, ns,k and hp,*, are used and every sk is 
25 initialized with 1.0. After every transmission of a frame, 
its expected error is updated and all the dependent 
priorities within the same GOF have to be recalculated. 

Even though the embodiments have been described as individual 
3 0 examples, it is explicitly noted that some or all of the 
embodiments may be combined, as desired. 

Further, a program may be provided having instructions 
adapted to carry out any of the operations of the above 
3 5 methods. 
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A computer readable medium may embody the program arid, a 
computer program product may comprise the computer readable 
medium . 

The above embodiments illustrate aspects of the invention for 
prioritising data elements of a data stream for transmission 
to a receiving devi luding determining a clecodability 

of data elements at a receiving device, and for prioritising 

the data elements based on the decodability and further 
factors, such as an overall decodability increase achievable 

by transmitting a data element, and further: under 
consideration of an error-prone data transmission medium. The 

above embodiments outline assigning priority values to data 
elements, e.g. for streaming video files, particularly over a 
transmission medium with variable bandwidth, if bandwidth 
requirements may not be met. The priority values can be 
generated from the structure of the stream which can then be 
used by appropriate scheduling algorithms to drop the least 
important parts of the video and optimise the decodability of 

that stream. 



